Display capable of interacting with an object

ABSTRACT

A system includes a screen and a computer that detects an object near the screen and allows interaction between the object and the computer.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.17/073,491, filed Oct. 19, 2020, which is a continuation of U.S.application Ser. No. 15/207,841, filed Jul. 12, 2016 (now U.S. Pat. No.10,809,854), which is a continuation of PCT/US2015/010838, filed Jan. 9,2015, which claims priority under 35 U.S.C. § 119(e) to U.S. ProvisionalApplication No. 61/926,845, filed on Jan. 13, 2014, and to U.S.Provisional Application No. 61/987,341, filed on May 1, 2014, the entirecontents of each are incorporated herein by its reference.

BACKGROUND 1. Field

Embodiments are directed to a recognizing an object on or near a displayscreen.

2. Description of the Related Art

Object recognition has been realized on a touch screen, e.g., using atouch screen that includes a camera or optical sensor embedded in thetouch screen. With previous approaches, a bar code or a label with aparticular pattern is placed on the bottom surface of an object. Theobject is placed on the table and the camera is used to read the patternon the label. However, hardware for these types of touch screens is moreexpensive than other hardware, e.g. PCT (Project Capacitive Technology),which can detect touch, but cannot detect optical patterns on a label onan object, and non-touch screen technology.

SUMMARY

One or more embodiments is directed to a system, including a screen, anda computer that detects an object near the screen through use of acamera on the object without using a visible code on the object and thatsets up a dedicated link between the object and the computer.

One or more embodiments is directed to a system, including a largeformat display, and a host computer in communication with the largeformat display, wherein, when an object including a camera is near thelarge format display, the host computer displays codes on the largeformat display for detection by the camera, to establish a dedicatedlink with the object based on the codes detected by the camera, the hostcomputer to determine a location of the object.

The system may include an interactive sensor.

The host computer may determine a location of the object in response toa touch by a user adjacent the object.

The host computer may display a representation of the object in a regionadjacent the object.

When the object includes an interactive sensor, items dragged off of theobject towards a border of the representation may be sent to the largeformat display.

Items dragged from the large format display towards the representationmay be sent to the object.

The host computer may be configured to run a device present code acrossthe large format display and determine whether the camera detects thedevice present code. The device present code may be invisible to viewersof the large format display. The device present code may include varyingat least one of color and brightness by a predetermined amount for apreset block of pixels. The device present code may include varying atleast one of color and brightness by a predetermined amount for a presetblock of pixels over specific time intervals.

Once a device is determined to be present due to detection of the devicepresent code by the camera, the host computer may run a code withposition identifiers across the large format display and determinewhether the camera detects the code with position identifiers.

The host computer is configured to run the code with positionidentifiers across increasing smaller regions of the large formatdisplay until the position of the object is determined.

If the code with position identifiers includes more than one code blockpixel, the host computer may extrapolate an orientation of the object onthe large format display.

Once the position of the object is determined, the host computer may runa code with orientation identifiers at the position of the object todetermine an orientation of the object on the large format display.

The large format display may be oriented horizontally. The object may beplaced on the large format display.

The host computer may not use a physical unique identifier on theobject.

The host computer may not use an optical sensor to determine thelocation of the object.

The view of the camera may not be displayed on the object's screen.

The host computer may establish the dedicated link automatically.

The location of the object may be directly on the large format display.The location may be on a specific region of the large format display.The specific region of the large format display may be predefined oruser defined. The specific region of the large format display may bedetermined by the host computer. The host computer may display arepresentation of the object on the large format display in the specificregion.

The location of the object may be adjacent the large format display. Thelocation may include a specific region of the large format display. Thespecific region of the large format display may be predefined or may beuser defined. The specific region of the large format display may bedetermined by the host computer. The host computer may display arepresentation of the object on the large format display in the specificregion.

The host computer may display a region on the large format display towhich items to be transferred between the large format display and theobject are to be placed. The object may be used to change content of theregion. The region may be adjacent the object.

The host computer may run a code with position identifiers across thelarge format display to determine the location of the object.

One or more embodiments is directed to a system including a large formatdisplay, at least two wireless transmitters, and a host computer incommunication with the large format display, wherein, when an objecthaving a wireless receiver in communication with the at least twowireless transmitters is near the large format display, the hostcomputer uses relative signal strength from the at least two wirelesstransmitters detected by the wireless transmitters in the object todetermine a location of the object.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1 and 2 illustrate a top schematic view of a table displayproviding a code to a mobile device according to embodiments;

FIG. 3 illustrates a flow chart of a method from the point of view of adisplay computer according to an embodiment;

FIGS. 4 to 10 illustrate schematic plan view of the display and themobile device interacting;

FIG. 11 illustrates a flow chart of a method from the point of view of amobile device according to an embodiment;

FIG. 12 illustrates a configuration according to an embodiment; and

FIG. 13 illustrates a configuration according to an embodiment.

DETAILED DESCRIPTION

Embodiments are directed to a method of establishing communicationbetween a mobile device, e.g., a smart phone, a tablet, and so forth,and a large format (LF) screen so that information on the mobile deviceor associated with the mobile device can be displayed on the LF screenand information on the LF screen can be transferred to the mobiledevice. The mobile device may include an interactive sensor, e.g., atouch sensor, a gesture sensor, and so forth. The LF screen may beembedded in table or a kiosk or placed on a wall of a stand. Suchcommunication needs a communication link to be established between themobile device and a display computer driving the LF screen.

In a single person mode, the link may be established in manner so thatthe person using the LF screen is the owner of the mobile device that isconnected to the LF screen during that person's session. In amulti-person mode, the link may be established in manner so that theperson using that person's section of the screen is the owner of themobile device that is connected to the corresponding section of thescreen that the person is using. Single person mode as used herein is tomean only one person is associated with the use of each LF screen. Moreparticularly, one mobile device is associated with each LF screen. Inmulti-person mode, multiple mobile devices are associated with an LFscreen. So in multi-person mode, there can be communication linksestablished for a single LF screen with multiple mobile devices.

One way to establish a communication link is using passwords asdescribed in U.S. Patent Application Publication No. 2012/0162351 A1,which is incorporated by reference herein in its entirety. When a userdesires to interact with an LF screen the screen could display apassword or code for the user to type in to their mobile device.Alternatively, the password could be displayed on the mobile device andthe user could type the password in on a virtual keyboard displayed onthe LF screen if the LF screen is a touch screen. However, many wouldprefer to interact with a system without having to remember/inputpasswords. Thus, one or more embodiments is directed to establishinglinks without the user having to enter passwords for each session.

Typically there will be a Large Format (LF) screen 210, with or withoutan interactive sensor 240, e.g., a touch sensor, a gesture detector andso forth, embedded in a table 220, and a display computer 230, asillustrated in FIGS. 12 and 13 . Additional elements, e.g., a piece ofcover glass, a diffuser, a structural mechanical mount, and so forth,may be provided above or below the interactive sensor 240, e.g., betweeninteractive sensor 240 and LF screen 210.

One alternative to continuously entering passwords is the use of a barcode, e.g., a quick response (QR) code, as shown in FIG. 1 . When a QRcode is displayed on the table display 14, a camera in the mobile devicecan take a picture of the QR code. Reader applications for QR codes areubiquitous. The QR code detected by the reader application on the mobiledevice then directs the user to a specified website. In particular, eachQR code could be unique to each table, to each request, and/or eachsession. Each unique QR code could take the user to a different webaddress. Every time a new QR code is read, the mobile device would bedirected to a unique URL address which can act as a key (embeddedpassword) for the display computer 230. Then, when the display computer230 detects a device trying to access it, with the correct key, thedisplay computer 230 determines that the device just received the QRcode. This website could obtain a key from the mobile device. Then, thedisplay computer 230 can note a particular characteristic of the mobiledevice, e.g., a MAC address, an IP address, a device name, and so forth.Once this happens, the display computer 230 can instruct the user toshare information with the table system. The user can then follow theinstructions to share content with the display computer 230. Content maybe shared either with using built in features of the operating systemsof the mobile device, e.g., Airplay with iOS. When the user tries tostream information to the display computer 230, the display computer 230will check the MAC address or ip address of the mobile device to makesure it is valid. If so, the display computer 230 may then allow contentto be shared.

Another alternative when using a bar code or QR code is to use aninteractive application (IA) loaded on a mobile device 300. The IA candigitize output from a mobile device and stream it to the displaycomputer 230, or utilize the mobile devices OS to stream the contents ofthe mobile device's screen as described in U.S. Patent ApplicationPublication No. 2013/0219295 A1, which is incorporated by referenceherein in its entirety. By using the IA, security can be very robust andnot be dependent on an operating system of a particular mobile device.The QR code could automatically direct the user to a website that wouldask whether the user wants to download the IA. Alternatively,instructions, including a website from which the IA can be downloaded,could be displayed, e.g., on the table display 14. Once the IA isdownloaded or otherwise on the mobile device, a QR code would bedisplayed again, either the same or a new QR code, to obtain the keyinformation.

As an alternative to using a QR code, a camera of the mobile device maydetect a digital watermark, as shown in FIG. 2 . This digital watermarkis similar to a QR code in that it gives a digital code back to the IA,but the digital watermark is imbedded in the LF display and is notvisible to the user. The application sends this code to a server alongwith the username password, which may be encrypted. Once the serverreceives this code, the server knows exactly which table and whichsession on that table to send the username and password to, e.g., the QRcode may be unique for each table and session. This allows multiplepeople to login to user sessions on the same display or multiple peopleusing displays that are located very close to each other. The digitalwatermark could be embedded in a particular image (“WM Image”), e.g.logo, or dedicated region, as moving or changing images are harder tofocus on. In this case, typically the mobile device would not be able tobe placed directly on the table or display. Typically, the user wouldhold the device and look through the camera in order to focus on the WMImage. Once the WM Image is in focus, then the IA would grab the imageand decode it.

Note that in both cases (use of the QR code and WM Image), a uniquepassword or code is associated with QR code/WM Image. In single personmode, this unique password/code would therefore be associated with aparticular LF screen. Once the QR code or WM Image is scanned, theunique password/code is also associated with a particular mobile deviceand there can now be a dedicated communication link established betweenthe mobile device and the LF screen, i.e., nearby users would not beable to send/receive information from the LF screen. (Host computerknows which mobile device to receive and send information to.)Similarly, in multi-person mode, the dedicated link is established witha mobile device and a section of the LF screen.

Note that for multiperson mode, as described in U.S. Pat. No. 8,583,451B2, which is incorporated by reference herein in its entirety, thescreen may be divided in to sections with clear boundaries, for eachuser. A QR code or WM Image could be displayed in one of these sectionsto associate that section with a particular user/mobile device. Once thecommunication link is established in this manner, information from themobile device could be obtained from the host computer including forexample a user name. The user name could then be displayed in thesection of the LF screen corresponding to the mobile device (section forwhich the mobile device scanned the QR code or WM Image). If the LFscreen is a touch screen, then sections of the screen could then berearranged on the LF screen (e.g. change in position and size) with thecommunication links preserved with a visual indication (such as displayof the user name), such that each section could be associated with adifferent user/mobile device.

Another way to handle multiple users connected to a single LF screen, isto display the user names in a particular section of the screen (UserName Section). Then when data is required to be exchanged in aparticular section of the LF screen, a user name can be chosen from theUser Name Section. For example, suppose a pdf file is chosen on the LFscreen and an option on the LF screen is chosen to email this file out.An email window could then be displayed on the LF screen and one of thenames in the User Name Section could then be dragged to the emailwindow. Once this occurs, the information in this window would then beassociated with the user corresponding to the name dragged in thismanner (e.g. email address, name, company name, etc.)

One potential disadvantage with these approaches is the mobile devicemay need to be held away from the LF display in order to bring the QRcode or the WM Image into focus. Typically, these approaches alsorequire the user to view the camera's output and adjust the position tobring the QR code or the WM Image in to focus. This is satisfactory foruse with an LF screen that is not positioned such that the mobile devicecould be placed thereon, e.g., a wall mounted LF screen. However, itwould be faster and more convenient to place the mobile device directlyon the LF screen and enable the handshaking to occur automatically,e.g., passively, i.e., without any action on the part of the user. It isalso desirable to, on the mobile device's touch screen, displayinstructions and/or graphics associated with the transaction takingplace that may be different from the graphics displayed on the LFscreen. It is often further desirable to display this instead ofdisplaying the output from the camera.

An alternative to using the QR code or the WM Image is to use locationinformation embedded in each mobile device. If the mobile device isrunning the IA, then the host computer can access the GPS information.This typically will be more useful in single person mode. If the GPSinformation is accurate enough, and there is only one mobile devicewithin a designated distance of the display computer, then the displaycomputer can assume that this mobile device is the one to be connectedto the LF screen. To further ensure that the correct mobile device isconnected, once a connection is needed to be established, for examplewhen a password or email device would be typically requested on the LFscreen, a Connection Request Message could be displayed on either the LFscreen or the mobile device. For example, if it is displayed on themobile device it could ask if the user would like to establish aconnection with the particular LF screen or table/furniture. If thedisplay computer detected multiple devices within range of is associatedLF screen, it could send this request to all of the devices within aspecified range. Alternatively, if the Connection Request Message (CRM)is displayed on the LF screen it could list all devices within range onthe LF screen (or user names associated with each of them) and the usercould select their name from the list to establish a connection.

From this point on, as long as this mobile device remains within aspecified region from the LF screen from the GPS coordinates, thencommunication between the display computer and the mobile device cantake place.

For multi-person mode, distinct regions can be displayed on the LFscreen as described previously. These regions could be eitherpre-determined or user-defined regions. If multiple devices are detectedwithin range of a table and the names displayed as described above,different names could be dragged in to different regions, to associatedspecific users with specific sections of the LF screen. If CRM's aresent to multiple devices within range of an LF screen and more than oneof these are accepted by the user, then the names for only those thatare accepted could be displayed on the LF screen and the individualnames could then be dragged to specific regions of the LF screen.

However, currently GPS information from mobile devices is not accurateenough for this purpose for many applications. Current GPS accuracy inmobile devices is limited to about 50 ft to 100 feet and less in hightraffic areas. GPS may be used to determine, for example, which mobiledevices are in a building, but probably not which are within a few feetof a table, let alone on the table.

Another option would be to use wireless transmitters, e.g., BluetoothLow Energy (BLE) transceivers. Through triangulation and using multiplewireless transmitters, the accuracy to within 10 to 20 feet, which maybe accurate enough in some cases. The signals from these wirelesstransmitters would be detected by wireless receivers on the mobiledevice 300.

Yet another option would be to place the wireless transmitters inmultiple locations around the periphery of the LF screen. In this case,by looking at the difference between signal strengths detected by theBLE receivers on the mobile device 300, one can determine whether or notthe mobile device is inside a region formed by a geometric perimeterformed by the locations of the BLE transmitters. If the GPS or BLEmethods described above are not accurate enough to detect the locationof the mobile device, then a code method described below may be used.

Designated Regions

In addition to determining which device to associate with a particularLF screen or screen section, it is sometimes further desirable to alsohave the display computer know more precisely where on the LF screen aparticular phone is located. This may be achieved by providingdesignated areas on the LF screen on which a user is to place the mobiledevice or hold the mobile device adjacent thereto. These designatedareas could either be predetermined or use-defined.

For example, a predetermined region could be indicated on the LF screenas where a user can place a mobile device or hold the mobile deviceadjacent thereto. When a password is needed, the LF screen could displaytext, e.g., “place mobile device here to connect to this screen.”Alternatively, a user may place the mobile device in contact with orhold the mobile device adjacent to the LF screen having a touch screenembedded therein and indicate, e.g., outline, circle, and so forth, themobile device, to define a designated region, as illustrated in FIG. 5 .

Once the mobile device is placed on the designated region or thedesignated region is defined by the user, then as long as the displaycomputer can detect which mobile device is on the screen, the hostcomputer would know where on the LF screen the mobile device is located.Information can be dragged to the mobile device from the LF screen oroff of the mobile device onto the LF screen as described below.

Direct Contact Camera Codes (DCCC)

As described previously, in some cases BLE and/or GPS may not beaccurate enough to determine which mobile devices to connect to aparticular LF screen. In this case a Direct Contact Camera Code (DCCC)can be used. DCCC codes have multiple purposes: (1) they can be used asan alternative or to supplement the methods described above to determinewhich mobile device to associate with an LF screen, (2) they can be usedto determine the exact location of a mobile device on an LF screenand/or (3) once a communication link is established, the mobile phonecan be moved on the screen and the precise location of the device on thescreen can still be determined.

Once the mobile device is in the predetermined or user defineddesignated region, then a DCCC code can be displayed within thedesignated region only. In this case, simpler codes can be used ascompared with a digital watermark, since the need to have the code beinvisible to the user is not as necessary, since the majority of thedesignated region may be covered by the mobile device. The table display14 may display a time sequential code, e.g., a series of colors, withinthe circle. Just as in the case of the QR code method and the WM Imagemethod, a unique key, password, or other information can be embedded inthe DCCC. The mobile device detects the DCCC code and then tries toconnect to the display computer using the information embedded in theDCCC. Since the display computer will know where the mobile device isand the physical location will block the image being displayed on the LFdisplay from users, the code may be simpler and easier to detect by thecamera of the mobile device, as the code would no longer have to beinvisible to users.

In multi-person mode, a first person can place their mobile device in afirst designated region. Once this mobile device is detected andconnected, a second user can be directed to place their mobile device ina second designated region. The multiple devices can be detectedsimultaneously if different information is embedded in the differentDCCCs to designate the particular region within the LF screen.

If designated regions are not to be used, i.e., a user may place amobile device anywhere on a LF screen, the following approach may beused. As described below, DCCC codes displayed on the LF screen may beused to determine the precise location of the camera and orientation ofthe mobile device as well as the perimeter of the mobile device.

FIG. 3 illustrates a flow chart of a method from the point of view of adisplay computer according to an embodiment. As illustrated in FIG. 3 ,rather than an indicator physically on an object near a display, aninteractive application (IA) loaded on a mobile device 300 uses thecamera function in the mobile device for object recognition, e.g., toestablish a dedicated link between the mobile device and a host computerassociated with the display, and then allows the mobile device tointeract with the display. Typically there will be the Large Format (LF)screen 210, with or without the interactive sensor 240, embedded in thetable 220, and the host computer 230, as illustrated in FIGS. 12 and 13.

Initially, the user could place their device on the LF screen with itstouch screen face up. In operation S110, a user is instructed to run (ordownload and run) the IA and place their mobile device (e.g. smartphone, tablet, etc.) on the LF screen. Typically, the mobile devicewould be placed face up on the LF screen, so that the screen on thedevice can still be viewed by the user. However, if placed face down,the IA could detect this and make sure the camera has a correctorientation for the mobile device to be recognized. The instruction torun could be provided by the IA on the mobile device and/or by thedisplay computer on the LF screen when the display computer detects aunconnected mobile device within a certain range. Alternatively oradditionally, instructions could be posted near the LF display. As afurther alternative, when the display computer detects a mobile devicehaving the IA on it, the IA could automatically run on the mobiledevice. Once placed on the LF display, the IA may force that the cameraof the device to be oriented properly towards the LF screen or maydetect which camera is oriented towards the screen. When running the IA,the mobile device screen may indicate various modes of operation:detected device, downloading info, etc. For example, a first mode couldbe “searching;” a second mode could be “location of device on screenfound;” and a third mode could be “sharing,” etc. In this manner, the IAcan activate the camera on the object for the camera to detect the code,but since there is no need for the camera to focus on the code, there isno need for the camera on the object to display the view of the cameraon the object's touch screen.

Once placed on the LF screen, the LF screen can display the DCCC codeddata in various sections of the LF screen to detect the location of thedevice on the screen in operation S120. For example, the LF screen couldbe divided, e.g., into four quadrants, with each quadrant “flashing” acode in a particular color in a specific time sequence. The deviceapplication may be set to find this particular sequence in the firstmode, e.g., searching mode.

When a device is placed directly on the LF screen, only a very smallsection of the LF screen is visible to the camera of the device, e.g., aregion of about 1-2 mm in diameter up to about ½ inch wide, depending onthe distance from the camera to the display. The size of this sectionmay be controlled to some extent by increasing the distance between thetop of the upper surface of the table and the LCD display. Increasingthis distance helps increase the size of the area viewable by thecamera, but also degrades the image on the display, e.g., if there is adiffuser surface on a touch screen surface associated with the LFscreen, and increases the parallax issue with touch.

The net result is that typically only 1-2 mm of the display is visibleby the camera of the device. Depending on the camera in the device, theview may be quite blurry. Therefore, a useful code to be displayed maybe only a small number of effective pixels that can be read at a time,e.g. a 3×3 code block pixel, a 2×2 code block pixel, or only 1 effectivecode block pixel.

According to a first embodiment for operation S120, a 2×2 code blockpixel may be used. Within this 2×2 code block pixel, a color andbrightness of each of the four code block pixels may be considered andset to predetermined values. In turn, each of these code block pixelsmay be multiple pixels on the display, since the camera may not be infocus. The code may be changed with time.

The simplest method would be to use a black and white code and timesequence the code for about 10 iterations×4 which would give useffectively about 40 code block pixels. This code could be displayedover a first half of the LF screen and then over a second half of the LFscreen. If a device is detected on the first half of the LF screen, thatfirst half could be divided into, e.g., 2-4 regions and check there andsuccessively decrease the area until the mobile device is found. Thefirst code to be displayed could be a specific code identifying thespecific host computer driving the display.

A potential issue with this embodiment is that the user would see theDCCC codes being displayed on the screen. This may prevent otherinformation from being displayed and visible to the user while the DCCCcodes are being displayed.

According to another embodiment for operation S120, a visual image isdisplayed across the entire LF screen, and is varied only in a smallregion. The variation can be in color and/or brightness and timesequencing can again be used. For example, consider again the 2×2 codeblock pixels. Each of these regions would have specific values of colorand brightness across the display corresponding to the values of theimage displayed. In a specific 2×2 region one of these 4 pixels can beincreased in value by x % and another decreased in value by x %, etc.After a specific time interval, this increase and decrease can bechanged. The application can then search for changes in brightnessacross the display in the specific pattern in time and space thatmatches the code.

In this case, images or videos can be displayed on the LF screen as innormal operation, and the first DCCC code could be continuouslydisplayed across the display, even when no device is present, since itwould not be detectable to a user. When a device is placed on thedisplay running the app, then the above process could be started,without the need to place the screen in to a “seek device” mode. Thesame code of 40 code block pixels (4 in space and 10 iterations in time)could be superimposed over the entire screen at the same time and thisDCCC code would not be visible to the user.

Once the IA determines the particular sequence has been detected by thecamera of the device, the IA could then be moved to the second mode inoperation S130. For example, the display computer could also send asecond DCCC code across the entire screen, with each location specifyingthe particular location on the screen where the DCCC code is displayed.In this manner the device would also receive a code indicating itslocation on the LF screen.

Alternatively, the sections of the screen displaying the code could bemade smaller and smaller until the actual location of the device on thescreen is found in operation S140. Then the IA could then proceed to thethird mode.

In operation S150, orientation of the device on the LF screen may beobtained by using the data collected using the location information fromoperation S140. For example, the orientation of the DCCC codes foundduring operation S140, could be used to determine the orientation of thecamera on the LF screen, thereby determining the orientation of thedevice. Alternatively, once the location of the mobile device isdetected, a separate DCCC code could be used to determine theorientation of the device. Once operation S160 determines that theorientation information has been received, final verification of receiptof position and orientation may be confirmed in operation S170.

Once the device is detected, the IA and the display computer can thencommunicate through, e.g., wireless, Bluetooth, or other communicationprotocol in operation S180 via a dedicated, unique link.

There are numerous examples of data that may be exchanged or shared inthe third mode. If the venue is a retail site, the IA may automaticallylogin to an account of a website associated with the retail site. The IAcan use saved password and login name to login to store's website or toaccess information provided by user to the website or can auto login toa user's account, with saved information, including user preferences,user's history, payment information, and so forth.

If the venue is generic or unknown, a website may be displayed on the LFscreen 210 asking for a user's password and/or login name. The IA maydetect the website displayed and get info from the mobile device for theparticular website displayed on the LF screen.

If the venue is a restaurant, the mobile device may share paymentinformation, user account information, photos, videos or other media.

If the venue includes collaborative venues, e.g., libraries, schools,offices, the mobile device may share payment information, media, otherdocuments or files, user preferences, saved sessions, and so forth.

If the venue includes use management, e.g., reservation/wait list/linemanagement, if an LF display is reserved for a particular time, only amobile device with the correct activation code can use the display atthat time. The activation code could be sent to the user, or could beembedded in the app so it is not visible to the user. Here theinformation exchanged could be just the activation code, but would notrequire the user to enter the activation code. This could be used anyplace where you could make an appointment, make a reservation or wait ina line, e.g., government offices, medical offices, banks, airports,restaurants, and so forth.

Depending on the forum, e.g., a library, other research venue,restaurant, the display computer could save sessions associated with themobile device.

If the mobile device is moved, e.g., dragged, across the LF display, thedisplay computer should still be able to tell where the mobile device isby sending position identifiers in operation 130 to a small region,e.g., 0.5-1 inch for a predetermined period of time, surrounding wherethe mobile device was previously located. If the mobile device is notfound after the predetermined period of time, the mobile device would beconsidered lost and the method would restart at operation S110. If themobile device is picked up and moved such that the dedicated link isbroken, the method could start over at operation S110.

During re-establishing the dedicated link, e.g., anywhere betweenoperations S130 and S170, the method may ask permission as to whetherthe dedicated link is to be reestablished, e.g., by entering “OK” on themobile device or on the LF screen.

By detecting both the position and orientation of the device,information from the device may be displayed on the LF screen as if thatinformation is spilling out of the device and/or the display can displaysomething relative to the mobile device, e.g., the table computerreceives information from the mobile device, and then displays it on theLF screen.

For example, as illustrated in FIG. 4 , the display computer 230 coulddisplay a bubble 250 adjacent the mobile device 300 with an instructionas how to enable sharing of information on the mobile device with thedisplay computer 230. In the example shown in FIG. 4 , the user isinstructed to touch the activate button displayed on the mobile device.Once sharing is enabled, the content of the bubble 250 may be altered asappropriate. For example, as shown in FIG. 5 , a user may be prompted toselect payment information. If the system includes the interactivesensor 240, the user may indicate a region corresponding to the item tobe selected for detection by interactive sensor 240 and the interactivesensor 240 would then send the information to the display computer 230.If the system does not include the interactive sensor 240, but themobile device 300 does, the user may indicate a region corresponding tothe item to be selected on the interactive sensor of the mobile device300 and the mobile device 300 would then send the information to thedisplay computer 230. Using the mobile device to communicate with thedisplay computer 210 and change the content of the bubble could also beprovided as an option even when the interactive sensor 240 is present,or this redundant information may not be provided on the mobile devicewhen the interactive sensor 240 is present.

If the IA is part of a larger application dedicated, e.g., to aparticular venue, the IA may ask user whether to login at that venue.Such login may allow activity from a previous visit stored on thedisplay computer or in the cloud to be accessed by the user. Such logininformation may be stored on the mobile device and simply and readilyactivated by a user. For example, as illustrated in FIG. 6 , a logininstruction could be displayed in the bubble 250 on the display 210,with the actual login activation being performed on the mobile device,e.g., when the display 210 does not have the touch screen 240.Alternatively, when the display 210 includes a touch screen 240, asshown in FIG. 7 , the activation could occur on the display itself,e.g., in the bubble 250.

Also, as shown in FIG. 7 , the display 210 can display an outline 260 orother representation of the mobile device. The IA can also access themodel number, and hence the dimensions of the mobile device on which itis running. As the mobile device is dragged around the LF screen, theoutline 260 may move with the mobile device.

This outline 260 may remain on the display 210 even when the mobiledevice has been removed, as illustrated in FIG. 8 . Thus, while theabove has assumed the mobile device is to be placed on a horizontalscreen, a mobile device could interact with a display having otherorientations by just placing the mobile device against the display whileconnecting with the display computer, and then, once removed, ratherthan being disconnected as discussed above, the outline 260 may remainuntil, e.g., logout, time out, or other disconnect event occurs.

Additionally, as illustrated in FIG. 9 , files 310 from the mobiledevice 300 may be displayed on the display 210, e.g., by draggingparticular file off the mobile device, tapping on the file, tossing thefile, e.g., as disclosed in U.S. Pat. No. 8,522,153, which is herebyincorporated by reference in its entirety.

Additionally, virtual objects could be manipulated on the LF screenrelative to the mobile devices thereon. For example, as illustrated inFIG. 10 , a virtual bill 270 could appear on the LF screen to instruct auser where to drag information from the mobile device for a particularactivity, e.g., selecting a payment method displayed on the mobiledevice. Then, when the LF screen is a LF touch screen, the entire billcould be dragged to a particular mobile device or different items fromthe bill could be dragged to different mobile devices. Once the bill orparticular items have been allocated, the user may then pay using themobile device. The mobile devices could be physically on the LF touchscreen or they could be virtual mobile devices, if the devices have beenremoved and their outlines or virtual images remain as described above.Additionally or alternatively, the user can drag items on the mobiledevice off of the mobile device onto the LF screen or onto virtualobjects on the LF screen, even if the LF screen is not touchable. Forexample, methods of payments may be displayed on the mobile device orinformation for logging in may be displayed on the mobile device.

Using this contact embodiment may simplify determination of where thecontent came from when multiple devices are supplying content, as wellas simpler detection when the mobile device is removed. Once the mobiledevice is removed from the table, pictures/data associated with thatdevice can also be removed instantly or after a delay, eitherpredetermined or user selected.

Alternatively to the downloading of the IA on the mobile device, a webapplication may be employed to perform the operations of FIG. 3 . Forexample, a QR code may be provided on or near the display that, onceread by the mobile device, loads a website onto the mobile device.

FIG. 11 illustrates a flow chart of a method from the point of view of amobile device according to an embodiment. In operation S410, a userstarts the application on the mobile device and places the mobile deviceon the LF display. In operation S420, the mobile device indicates thehost computer is searching for the location of the mobile device. Inoperation S430, the host computer asks the user of the mobile devicewhether the user wants to allow sharing of information.

By way of summation and review, one or more embodiments is directed todetecting objects placed on or near a display using technology that doesnot use optical sensors and does not require a unique physicalidentifier, e.g., bar code, on the mobile device to interact with thedisplay. Further, one or more embodiments is directed to detecting anobject through use of a camera on the object without using a visiblecode on the object and set up a dedicated link.

Example embodiments have been disclosed herein, and although specificterms are employed, they are used and are to be interpreted in a genericand descriptive sense only and not for purpose of limitation. In someinstances, as would be apparent to one of ordinary skill in the art asof the filing of the present application, features, characteristics,and/or elements described in connection with a particular embodiment maybe used singly or in combination with features, characteristics, and/orelements described in connection with other embodiments unless otherwisespecifically indicated. Accordingly, it will be understood by those ofskill in the art that various changes in form and details may be madewithout departing from the spirit and scope of the present invention asset forth in the following claims.

What is claimed is:
 1. A system, comprising: a shared display; at leasttwo wireless transmitters or wireless receivers; and a host computer incommunication with the shared display, wherein, when an object having awireless receiver or a wireless transmitter that can send signals to ordetect signals from one of the at least two wireless transmitters orreceivers, the host computer uses relative signal strength between theobject and the at least two wireless transmitters or receivers in todetermine a location of the object, on condition that the object is nearthe shared display, the host computer sends a connection request messageto the object and, on condition that the object has an interactiveapplication stored therein, the connection request message automaticallyruns the interactive application on the object to connect the object tothe host computer or present a list on the object to be selected on thehost computer, and on condition that the object is selected from thelist on the host computer, automatically connect the object to the hostcomputer.
 2. The system according to claim 1, wherein, on condition thatmultiple objects are within range of the shared display, the hostcomputer is configured to provide distinct regions corresponding to eachobject on the shared display.
 3. The system according to claim 2,wherein the distinct regions are predetermined.
 4. The system accordingto claim 2, wherein the distinct regions are use-defined.
 5. The systemaccording to claim 2, wherein the host computer is further configured todisplay an interaction region adjacent to each distinct region on theshared display, the interaction region facilitating interaction of theobject corresponding to that distinct region and the shared display. 6.The system according to claim 5, wherein the interaction region includesoptions displayed on a display on the object with additionalinformation.
 7. The system according to claim 5, wherein the interactionregion includes instructions on how to interact with a display on theobject.
 8. The system according to claim 5, wherein the interactionregion includes options on the shared display to be selected on theshared display to control the object.
 9. The system according to claim5, wherein the interaction region includes instructions on how tointeract with the interaction region from the object.
 10. The systemaccording to claim 9, wherein the instructions include dragging an itemon a display of the object to the interaction region or dragging an itemfrom the shared display onto the object.
 11. The system according toclaim 1, wherein, on condition that the host computer and the object areconnected, the host computer requires a login, interactive applicationis configured to login to the host computer.
 12. The system according toclaim 1, wherein, on condition that the host computer and the object areconnected and the host computer is a known host computer that requires alogin, the interactive application is configured to login to known thehost computer.
 13. The system according to claim 1, wherein, oncondition that the host computer and the object are connected and thehost computer is a known host computer, the interactive application isconfigured to provide user preferences to the known host computer. 14.The system according to claim 1, wherein, on condition that the hostcomputer and the object are connected and the host computer is a knownhost computer, the interactive application is configured to providepayment information to the known host computer.
 15. The system accordingto claim 1, wherein, on condition that the host computer and the objectare connected and the host computer is a known host computer, theinteractive application is configured to provide saved sessions to theknown host computer.
 16. The system according to claim 1, wherein, oncondition that the host computer and the object are connected and thehost computer is a known host computer, the interactive application isconfigured to provide user history to the known host computer.
 17. Thesystem according to claim 1, wherein the connection request messageincludes an activation code that correspond to a particular time for theobject to be connected to the shared display.
 18. The system accordingto claim 1, wherein, on condition that multiple objects are within rangeof the shared display, the host computer is configured to send theconnection request message to all objects detected.